tests: Stabilize screenshot tests and clean up flaky infrastructure#1220
tests: Stabilize screenshot tests and clean up flaky infrastructure#1220jderochervlk wants to merge 6 commits intomasterfrom
Conversation
jderochervlk
changed the title
jderochervlk
force-pushed
the
vlk/disable-animations
branch
from
9e967c5 to
dc90ba5
Compare
| run: yarn playwright install --with-deps | ||
| - name: Vitest | ||
| run: yarn ci:test | ||
| - name: Commit and Push changes |
There was a problem hiding this comment.
This was a patch to keep the files in sync across local runs and CI to make sure that the CI versions were always used so they would be the same, but now they are the same in headless mode as they are running on chrome so this isn't needed.
Updates the Markdown image caption test to use a new caption and image. Regenerates all related test screenshots to reflect the change.
Add detailed instructions for running and updating Vitest browser-based unit tests in the README. Remove outdated screenshot baseline PNGs from __tests__/__screenshots__.
Add guidance in README to be selective when updating screenshots.
Cloudflare deploymentDeployement ID: 288934cf-ac6f-4509-a062-c716bedba4e4 ⛅️ wrangler 4.63.0 (update available 4.80.0) ✨ Uploading _redirects |
1 participant
Fixes several issues with screenshot tests producing inconsistent or incorrect results.
I was getting different results running the tests locally versus the runs on CI, so I made some config changes to lock in the browser size and viewport. There were also some other issues I addressed to produce better and more stable results.
Key changes:**
Removed ~60 stale/flaky screenshot baselines - The bulk of this PR deletes a large number of
.pngscreenshot files from__tests__/__screenshots__/across many test suites (Button, CodeExample, Blog, DocsLayout, Footer, SyntaxLookup, SidebarLayout, etc.). These were presumably unreliable or no longer needed.Stabilized screenshot comparison config (
vitest.config.mjs) - Added explicit screenshot configuration:deviceScaleFactor: 1to prevent DPI-related differences across machinesscale: "css"for screenshot optionsthreshold: 0.2,allowedMismatchedPixelRatio: 0.05) to reduce false negativesui: false)Added test-specific CSS overrides (
styles/test-overrides.css) - A new stylesheet imported only invitest.setup.mjsthat:scrollbar-width: none) so scrollbar rendering differences across environments don't break screenshotsFixed NavbarSecondary tests - Switched from
<BrowserRouter>to<MemoryRouter initialEntries=["/docs/manual/introduction"]>, giving the tests a deterministic URL context rather than relying on the browser's actual location.Removed video screenshot assertion - In
MarkdownComponents_.test.res, thetoMatchScreenshot("markdown-video")call was removed (video rendering is inherently non-deterministic for screenshots).Removed auto-commit of screenshots from CI (
.github/workflows/pull-request.yml) - Thegit-auto-commit-actionstep that automatically committed updated screenshots on PRs was removed, preventing confusing auto-pushed changes in PRs.Updated
AGENTS.md- Added a rule clarifying that.jsxfiles should never be edited directly since they're compiler-generated by ReScript.Overall: This is a testing infrastructure cleanup focused on making screenshot tests reliable and deterministic across different environments (local dev, CI, headless vs headed browsers) by eliminating the most common sources of visual diff noise.